Techniques for Delayed Binding of Monitoring Mechanisms to Application-Specific Instrumentation Points

Online interaction with computer systems and applications allows developers to monitor, experiment with, and debug long-running, resource-intensive applications at runtime. Traditionally, developers statically bind a monitoring mechanism to each application-specific instrumentation point. This approach has shortcomings for online, interactive monitoring. Namely, static binding limits portability among monitoring systems; it may mismatch monitoring mechanisms to interactive requests for monitoring data; and, predictions for the performance and execution paths of instrumentation for static bindings are left to the developer. To address these concerns, we have created a new technique called monitoring assertions that allows monitoring systems to delay binding of monitoring mechanisms to application-specific instrumentation points until runtime. Our empirical results show that we can alter the performance of both the application and the monitoring system by removing static binding requirement of application-specific monitoring systems. 1. Interactive monitoring systems Interactive monitoring systems permit users to interact with the monitoring system during the execution of the target application. Users can enable, disable, and reconfigure the monitoring system to suit their exploration of the application [19, 17, 9, 6]. These interactive monitoring systems complement other forms of monitoring that use post-mortem analysis [25, 18, 16, 2, 3], but they share with such systems the need to analyze and display monitoring data [20, 11, 12, 4]. The interactive monitoring systems NASA financially supported Vetter with a Graduate Student Researchers Program Fellowship while he was a Ph.D. candidate at Georgia Tech. This work was also funded, in part, by NSF equipment grants CDA-9501637, CDA-9422033, and ECS-9411846. investigated by our group are those that permit end-users to capture application-specific information from targets, such as the position and velocity of molecules, or values of a solution matrix in a multi-grid solver. Some online monitoring systems, which include Paradyn, Quartz, and EEL [17, 2, 3], use modified compilers or executable editing to interrogate the application for performance-relevant information. However, we do not employ these techniques because they are limited in the types of information they can provide to the user [19], especially when executables are highly-optimized [7, 24]. Alternatively, users of application-specific monitoring systems annotate their application source code to supplement information available from these other monitoring methods. Our particular form of application-specific monitoring, called event-based monitoring [13, 4, 14], produces streams of events that represent observed application state; the user analyzes these streams judge target system behavior. Motivation. Application instrumentation is a primary component of any application-specific monitoring system. Across a wide variety of systems [23, 9], instrumentation support may be described as follows. (1) The developer chooses basic instrumentation mechanisms and annotates the source code. (2) At runtime, the monitoring system receives data from this instrumentation. (3) The monitoring server has limited runtime control over the instrumentation such that it typically relies on instrumentation placement to produce interesting data at appropriate rates. This static binding of monitoring mechanisms to instrumentationpoints is unacceptable for interactive monitoring systems because it is not easy to reasonably predict the interactor’s exploration scenario when deciding upon instrumentation mechanisms. The developer must also predict the interactor’s exploration scenario to help maintain consistent (or meaningful) data views of the target application. This paper introduces monitoring assertions that allow developers to instrument applications without forcing them to predict how monitoring data might later be used and without forcing them to judge a-priori which mechanisms are most appropriate. These assertions annotate what application data is available and when it is accessible to the monitoring system. Related work. Monitoring assertions are code annotations; we propose these assertions could be used by any monitoring system in place of their application-specific instrumentation. Then, the monitoring system would query and control the instrumentation using the ’monitoring assertions’ library API. Our earlier annotated bibliography provides many references to these instrumentation systems [10]. Tools like gprof[8] and Quartz[2] provide the user with valuable information about execution frequency; they assign elapsed time to static, syntactic units, such as procedures or statements. Additional work by Ammons and colleagues [1] extends this work to use flow and context information for performance counters. Paradyn [17] provides similar control-based application performance information; it delays the insertion of instrumentation until requested by the interactor, whether it is a query driven by a human or by Paradyn’s performance consultant. These systems [8, 2, 1, 17] rely on information stored within the executable by the compiler—either the instrumentation itself or instrumentation targets such as procedure entry points. In many cases, this limitation prohibits the use of executable editing to observe application-specific data. An example of this shortcoming is described in x3. Application-specific monitoring systems such as Falcon[9] and Autopilot [21] provide their own instrumentation libraries and instrumentation controls. Developers insert this instrumentation into their application source code and then, capture and analyze data produced by the instrumentation at runtime. Unfortunately, most every application-specific monitoring system has its own nonportable instrumentation library. In this respect, monitoring assertions could provide a common instrumentation tool for both application developers and tool developers. While these earlier instrumentation techniques propose various instrumentation mechanisms [19, 5, 16, 23], they focus primarily on the design of the monitoring system itself. Research contributions. The novel idea of this work is to generalize instrumentation so that the binding of specific monitoring mechanisms to instrumentation points in target applications is delayed until more is known about the expected use of the monitoring information (i.e., at runtime). Note that executable editing allows the monitoring system to delay binding until runtime; however, it then can only bind to a limited scope of application attributes, such as procedure entry points. Whereas, monitoring assertions allows developers to introduce as many application-specific binding points as necessary. (1) Monitoring assertions capture the desired properties of application-specific instrumentation of what data to monitor and when to monitor that data without the requirement of static binding to particular monitoring mechanisms. (2) Instrumentation signatures result in our system’s ability to predict monitoringoverheads and thereby enable runtime choices in instrumentation bindings based on observed patterns in instrumentation flow and frequencies. These signatures are derived from reference executions of instrumented applications. (3) Empirical evaluations of monitoring assertions and instrumentation signatures demonstrate the utilityof delaying mechanism binding until runtime. Sample applications. Two applications demonstrate our ideas. Heat diffusion is a 27-point, 3D-stencil, time-stepped simulation [22] implemented with kernel-level threads for SMP platforms. This simulation exhibits nearest neighbor sharing common to many simulations of physical systems. A more complex example is provided by the Splash 2 OCEAN benchmark. The Ocean benchmark simulates eddy and boundary currents in a cuboidal ocean basin. The application uses finite differencing CFD with a regular grid. The algorithm uses a red-black Gauss-Seidel multi-gridequation solver; each time-step of the simulation involves setting up and solving a set of spatial partial differential equations. Experiment platform. The experimental platform is a network of 4 two-processor Sun Ultra 2 Model 2148s (148 MHz UltraSPARC CPU). Each system has 128MB of main memory and they run Solaris 2.5.1. Although this work focuses on shared memory architectures, we are not aware of any fundamental design restrictions such that monitoring assertions could not work in a message passing environment. Paper outline. x2 reviews application instrumentation including various mechanisms and characteristics. x3 and x4 explain the technique of monitoring assertions and instrumentation signatures, respectively. x5 reveals our concluding thoughts and some future research issues. 2. Background: application instrumentation Typical online monitoring systems have four basic components as illustrated in Fig. 1: system software, application, monitoring server, and interactor. Systems that broadly fit this model include Paradyn [17], Falcon [9], Magellan [22], Avatar [20], Vista [23], and various debuggers. We assume no special operating system, special compilers, or hardware. The interactor controls the server; it specifically provides user interface and visualization capabilities. The server is generally a separate process or thread that communicates with the application via shared memory or IPC. Application-specific instrumentation is represented as software statements added to the target application [19, 5, 16]. This instrumentation gathers data about the application’s execution, packages it, and sends it to some higherlevel monitoringsystem. This instrumentationhas two basic objectives: (1) instrumentation identifies which application data Interactor supervises monitoring of application Monitoring server polls incoming interactor commands as well as data Certain monitoring mechanisms may buffered monitoring instrumentation Threads encounter instrumentation points